Method and apparatus for refining depth image

ABSTRACT

A method of refining a depth image includes extracting shading information of color pixels from a color image, and refining a depth image corresponding to the color image based on surface normal information of an object included in the shading information.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims the benefit under 35 USC 119(a) of Korean PatentApplication No. 10-2017-0044226 filed on Apr. 5, 2017, in the KoreanIntellectual Property Office, the entire disclosure of which isincorporated herein by reference for all purposes.

BACKGROUND 1. Field

This application relates to a technology for refining a depth image.

2. Description of Related Art

In recent days, a service using a depth image includingthree-dimensional (3D) depth information to be applied to, for example,game contents and medical contents, has been provided. In general, thedepth image is generated by additionally measuring a depth using anapparatus, for example, an infrared sensor and a stereo camera. Thedepth information indicates a distance between an object and ameasurement position. To obtain a desirable result using a depth image,the depth information included in the depth image may be maintained anda noise may be effectively removed from the depth image.

SUMMARY

This Summary is provided to introduce a selection of concepts in asimplified form that are further described below in the DetailedDescription. This Summary is not intended to identify key features oressential features of the claimed subject matter, nor is it intended tobe used as an aid in determining the scope of the claimed subjectmatter.

In one general aspect, a method of refining a depth image includesextracting shading information of color pixels from a color image; andrefining a depth image corresponding to the color image based on surfacenormal information of an object included in the shading information.

The refining of the depth image may include determining a filtercharacteristic of a filter to be applied to a current depth pixelincluded in the depth image based on a surface normal distribution ofthe surface normal information for each of regions in the depth image;and adjusting a depth value of the current depth pixel by applying afilter having the determined filter characteristic to the current depthpixel.

The determining of the filter characteristic may include determiningeither one or both of a filter coefficient of the filter to be appliedto the current depth pixel and a filter size of the filter to be appliedto the current depth pixel based on the surface normal information.

The refining of the depth image may include determining a type of aregion to which a current depth pixel included in the depth imagebelongs based on a surface normal distribution of the surface normalinformation in the region; and adjusting a depth value of the currentdepth pixel by applying a filter corresponding to the determined type ofthe region to the current depth pixel.

The filter may be configured to adjust the depth value of the currentdepth pixel based on a depth value of a neighboring depth pixel of thecurrent depth pixel.

The determining of the type of the region may include determining aregion to which the current depth pixel belongs based on a change ofsurface normal values of neighboring pixels of the current depth pixel.

The determining of the type of the region may include determiningwhether the current depth pixel belongs to a noise region, a surfaceregion of the object, or an edge region of the object.

A filter size of a filter corresponding to the noise region or thesurface region of the object may be greater than a filter size of afilter corresponding to the edge region of the object.

A filter coefficient to be applied to a neighboring depth pixel of thecurrent depth pixel may vary depending on whether the filter to beapplied to the current depth pixel is a filter corresponding to thenoise region, a filter corresponding to the surface region of theobject, or a filter corresponding to the edge region of the object.

The method may further include determining whether to refine a depthimage of a current time based on depth information of a depth image of aprevious time.

The method may further include refining the depth image of the currenttime based on the depth information of the depth image of the previoustime in response to a difference between a color image corresponding tothe depth image of the current time and a color image corresponding tothe depth image of the previous time satisfying a preset condition.

The method may further include extracting shading information of thecolor pixels based on depth information of the refined depth image.

The method may further include extracting albedo information of thecolor pixels from the color image; and the refining of the depth imagemay include refining the depth image based on a first weight based onthe surface normal information, a second weight based on the albedoinformation, and a third weight based on a difference between a colorimage of a current time and a color image of a previous time.

The shading information corresponds to a vector dot product between adirection of a light source and a surface normal of an object surface.

In another general aspect, a non-transitory computer-readable mediumstores instructions that, when executed by a processor, cause theprocessor to perform the method described above.

In another general aspect, a depth image refining apparatus includes aprocessor configured to extract shading information of color pixels froma color image, and refine a depth image corresponding to the color imagebased on surface normal information of an object included in the shadinginformation.

The processor may be further configured to determine a filtercharacteristic of a filter to be applied to a current depth pixelincluded in the depth image based on a surface normal distribution ofthe surface normal information for each of regions in the depth image,and adjust a depth value of the current depth pixel by applying a filterhaving the determined filter characteristic to the current depth pixel.

The processor may be further configured to determine a type of a regionto which a current depth pixel included in the depth image belongs basedon a surface normal distribution of the surface normal information ofthe region, and adjust a depth value of the current depth pixel byapplying a filter corresponding to the determined type of the region tothe current depth pixel.

The processor may be further configured to extract albedo information ofthe color pixels from the color image, and refine the depth image basedon a first weight based on the surface normal information, a secondweight based on the albedo information, and a third weight based on adifference between a color image of a current time and a color image ofa previous time.

In another general aspect, a method of refining a depth image, themethod includes determining a noise reducing method to be applied to adepth image based on surface normal information of an object in a colorimage corresponding to the depth image; and refining the depth image byapplying the determined noise reducing method to the depth image.

The noise reducing method may be a filter; the determining of the noisereducing method may include determining a filter characteristic of thefilter based on the surface normal information; and the refining of thedepth image may include applying the filter to a current depth pixel ofthe depth image.

The filter may be a spatial filter including a filter coefficient to beapplied to a depth value of the current depth pixel and filtercoefficients to be applied to depth values of neighboring depth pixelsof the current depth pixel to obtain a refined depth value of thecurrent depth pixel; and the determining of the filter characteristicmay include determining, based on the surface normal information, eitherone or both of a filter size of the spatial filter and a reduction rateat which the filter coefficients decrease from a center of the spatialfilter to a periphery of the spatial filter.

The determining of the filter characteristic may further includedetermining whether a region in which the current depth pixel is locatedis a noise region, a surface region of the object, or an edge region ofthe object based on a surface normal distribution of the surface normalinformation; and determining either one or both of the filter size andthe reduction rate based on the determined type of the region.

The determining of the filter size may include determining the filtersize to be a first filter size in response to the determined type of theregion being the noise region; determining the filter size to be asecond filter size smaller than the first filter size in response to thedetermined type of the region being the surface region of the object,and determining the filter size to be a third filter size smaller thanthe second filter size in response to the determined type of the regionbeing the edge region of the object; and the determining of thereduction rate may include determining the reduction rate to be a firstreduction rate in response to the determined type of the region beingthe noise region; determining the reduction rate to be a secondreduction rate greater than the first reduction rate in response to thedetermined type of the region being the surface region of the object,and determining the reduction rate to be a third reduction rate greaterthan the second reduction rate in response to the determined type of theregion being the edge region of the object.

The color image may be a color image of a current time; and the methodmay further include applying a temporal filter to the current depthpixel in response to a difference between a region of the color image ofthe current time corresponding to a region of the depth image in whichthe current depth pixel is located and a corresponding region of a colorimage of a previous time being less than a predetermined threshold.

Other features and aspects will be apparent from the following detaileddescription, the drawings, and the claims.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates an example of a system for refining a depth image.

FIG. 2 is a flowchart illustrating an example of an operation of amethod of refining a depth image.

FIG. 3 is a flowchart illustrating an example of a process of refining adepth image based on surface normal information.

FIG. 4 illustrates an example of a process of refining a depth image.

FIG. 5 illustrates another example of a process of refining a depthimage.

FIGS. 6A and 6B illustrate examples of determining a type of a region towhich a depth pixel belongs based on shading information.

FIGS. 7A and 7B illustrate examples of adjusting a depth value of adepth pixel using a filter.

FIG. 8 is a flowchart illustrating an example of a process of extractingshading information using feedback.

FIG. 9 is a flowchart illustrating another example of an operation of amethod of refining a depth image.

FIG. 10 illustrates an example of a configuration of a depth imagerefining apparatus.

Throughout the drawings and the detailed description, the same referencenumerals refer to the same elements. The drawings may not be to scale,and the relative size, proportions, and depiction of elements in thedrawings may be exaggerated for clarity, illustration, and convenience.

DETAILED DESCRIPTION

The following detailed description is provided to assist the reader ingaining a comprehensive understanding of the methods, apparatuses,and/or systems described herein. However, various changes,modifications, and equivalents of the methods, apparatuses, and/orsystems described herein will be apparent after an understanding of thedisclosure of this application. For example, the sequences of operationsdescribed herein are merely examples, and are not limited to those setforth herein, but may be changed as will be apparent after anunderstanding of the disclosure of this application, with the exceptionof operations necessarily occurring in a certain order. Also,descriptions of features that are known in the art may be omitted forincreased clarity and conciseness.

The features described herein may be embodied in different forms, andare not to be construed as being limited to the examples describedherein. Rather, the examples described herein have been provided merelyto illustrate some of the many possible ways of implementing themethods, apparatuses, and/or systems described herein that will beapparent after an understanding of the disclosure of this application.

Terms such as first, second, A, B, (a), and (b) may be used herein todescribe components. Such terms are not used to define an essence,order, or sequence of a corresponding component, but are used merely todistinguish the corresponding component from other another component.For example, a first component may also be referred to as a secondcomponent, and a second component may also be referred to as a firstcomponent.

If the specification states that one component is “connected,”“coupled,” or “joined” to another component, a third component may be“connected,” “coupled,” and “joined” between the first and secondcomponents, or the first component may be directly connected, coupled,or joined to the second component. However, if the specification statesthat one component is “directly connected” or “directly joined” toanother component, a third component cannot be “connected,” “coupled,”and “joined” between the first and second components. Similarexpressions, for example, “between” and “immediately between,” and“adjacent to” and “immediately adjacent to,” are also to be construed inthis same manner.

The terminology used herein is for the purpose of describing particularexamples only, and is not to be used to limit the scope of thedisclosure and the claims. The singular forms “a,” “an,” and “the,” areintended to include the plural forms as well, unless the context clearlyindicates otherwise. The terms “comprises,” “comprising,” “includes,”and “including” specify the presence of stated features, numbers,operations, elements, components, and combinations thereof, but do notpreclude the presence or addition of one or more other features,numbers, operations, elements, components, and combinations thereof.

Unless otherwise defined, all terms, including technical and scientificterms, used herein have the same meaning as commonly understood by oneof ordinary skill in the art to which this disclosure pertains based onan understanding of the disclosure of this application. Terms, such asthose defined in commonly used dictionaries, are to be interpreted ashaving a meaning that is consistent with their meaning in the context ofthe relevant art and the disclosure of this application, and are not tobe interpreted in an idealized or overly formal sense unless expresslyso defined herein.

FIG. 1 illustrates an example of a system for refining a depth image.

Referring to FIG. 1, the system includes an image sensor 110, a depthsensor 120, and a depth image refining apparatus 130. The image sensor110 obtains a color image (or color frame) 115 of a subject (or object).For example, the image sensor 110 may be a complementarymetal-oxide-semiconductor (CMOS) image sensor, a charge-coupled device(CCD) image sensor, or a stereo camera. The color image 115 includescolor pixels, and each of the color pixels has a color value. The depthsensor 120 obtains a depth image (or depth frame) 125 of a subject thatis the same as the subject of the color image 115. For example, thedepth sensor 120 is a Kinect, a time-of-flight (TOF) depth camera, or anoptical three-dimensional (3D) scanner. The depth image 125 correspondsto the color image 115 and includes depth pixels. Each of the depthpixels has a depth value indicating distance information of a subject.The color image 115 obtained by the image sensor 110 and the depth image125 obtained by the depth sensor 120 are provided to the depth imagerefining apparatus 130. The image sensor 110 and the depth sensor 120may respectively transmit a stream of color images, for example, thecolor image 115, and a stream of depth images, for example, the depthimage 125, to the depth image refining apparatus 130.

In an example in which a stereo camera is used as the image sensor 110,the stereo camera obtains a stereo image including a left image and aright image, and the obtained stereo image is provided to the depthimage refining apparatus 130. The depth image refining apparatus 130generates the depth image 125 from the stereo image based on awell-known stereo matching scheme. In this case, the depth imagerefining apparatus 130 obtains the color image 115 (left image and rightimage) and the depth image 125 from the stereo image without receivingthe depth image 125 from the separate depth sensor 120.

The depth image refining apparatus 130 refines the depth image 125 usingthe color image 115 and generates a refined depth image. The depth imagerefining apparatus 130 refines depth values of the depth image 125 togenerate the refined depth image. A process of refining the depth valuesincludes a process of reducing a noise component included in the depthimage 125. The depth image 125 may have a noise component, and it issignificant to effectively reduce a noise component while maintaining adepth characteristic, for example, a depth characteristic of an edge ofan object, included in the depth image 125 to obtain a desirable resultusing the depth image 125. For example, when implementing an augmentedreality (AR) capability, a 3D space modeling of a real-world image isneeded to combine the real-world image and a virtual object. The depthimage 125 is used for the 3D space modeling, and a noise component inthe depth image 125 may prevent a natural combination of the real-worldimage and the virtual object. In a process of refining the depth image125, characteristic portions, for example, an edge of an object, shouldbe maintained substantially unchanged. However, it is difficult todistinguish between an edge and a noise component in the depth image125.

As will be described in detail below, the depth image refining apparatus130 effectively refines the depth image 125 based on shading informationof the color image 115. The refining increases an accuracy of the depthinformation included in the depth image 125. The shading information isinformation of a shading value corresponding to a vector dot productbetween a direction of a light source and a surface normal of an objectsurface. The surface normal is a normal direction component of a surfaceof a 3D object. The depth image refining apparatus 130 distinguishesbetween a noise region and an edge region in the depth image 125 basedon a characteristic of a surface normal distribution represented in thecolor image 115, and effectively reduces a noise component whilemaintaining an edge characteristic substantially unchanged by applyingfilters having different characteristics to the noise region and theedge region in the process of refining the depth image 125.

A color appearing at a certain point of an object is affected by variousfactors, for example, a shape and a material of the object, a lightsource, and a viewpoint. The color image 115 including color informationof the object includes an albedo component (or reflectance component)and a shading component. The albedo component is a unique colorcomponent or a material characteristic of the object determined by theshape and material of the object. The albedo component is independent ofa light source and a viewpoint. The shading component is an illuminationcharacteristic value obtained when light emitted from the light sourceinteracts with a surface normal of the object. A change in a surfacenormal of the object may be estimated based on a change in the shadingcomponent, and whether a predetermined region in the depth image 125 isa noise region or an edge region of the object may be estimated based onthe change in the surface normal. The depth image refining apparatus 130removes a noise component from the depth image 125 by analyzing thesurface normal based on the shading component.

In one example, the refined depth image is displayed on a screen orstored in a storage device. In another example, the refined depth imageis transmitted to other computing devices. The refined depth image maybe used, for example, to implement a 3D user interface (UI), 3Dcontents, virtual reality (VR), or AR in an apparatus, for example, apersonal computer (PC), a laptop, a notebook, a netbook, a tablet, apersonal digital assistant (PDA), a navigation device, a home appliance,an image processing apparatus, a smartphone, a 3D television (TV), or adigital information display (DID). However, these are merely examples,and the refined depth image may be used in any device that uses a depthimage.

A detailed description of a method by which the depth image refiningapparatus 130 refines the depth image 125 is provided below.

FIG. 2 is a flowchart illustrating an example of an operation of amethod of refining a depth image.

The method of refining the depth image in FIG. 2 may be performed by thedepth image refining apparatus 130 of FIG. 1 or a depth image refiningapparatus of FIG. 10. Referring to FIG. 2, in operation 210, the depthimage refining apparatus extracts shading information of color pixelsfrom a color image. The shading information includes surface normalinformation indicating a surface normal component of an object surface.

In one example, a color of each of the color pixels included in thecolor image is determined by a product of a shading component and analbedo component corresponding to a unique color of an object. Thealbedo component and the shading component are separated from each otherbased on a characteristic that the albedo component changes sharply andthe shading component changes relatively smoothly in a color space. Forexample, when a color changes between neighboring color pixels, thealbedo component shows a discontinuity while the shading component showsa continuity. The albedo component and the shading component areextracted from the color image based on this characteristic. The shadingcomponent corresponds to a vector dot product between a direction of alight source and a surface normal of an object surface. If the directionof the light source is known or the direction of the light source isconstant in an entire region of the color image, a change in the surfacenormal component may be estimated based on a change in the shadingcomponent. After the shading component of a color pixel has beenextracted, the albedo component of the color pixel may be extracted bydividing the color of the color pixel by the shading component of thecolor pixel.

In operation 220, the depth image refining apparatus refines a depthimage corresponding to the color image. A process of refining the depthimage includes a process of reducing a noise while maximally maintainingan edge characteristic of the object in the depth image by filtering adepth value. The filtering may be performed based on depth informationof a neighboring depth pixel, or depth information included in a depthimage of a previous time.

In one example, the depth image refining apparatus analyzes a surfacenormal for each region in the depth image based on surface normalinformation of the object included in the shading information, andreduces a noise included in the depth image based on a result of theanalyzing. The depth image refining apparatus determines a type of aregion to which a current depth pixel belongs based on a gradient of thesurface normal information (or shading information).

For example, the depth image refining apparatus determines that acurrent depth pixel corresponding to a current color pixel is located ina flat surface region of an object if a gradient of the surface normalinformation between the current color pixel and neighboring color pixelsof the current color pixel is 0 or close to 0. If the gradient changesslowly between the color pixels, the current depth pixel is determinedto be located in a curved surface region of the object. If the gradientbetween the color pixels is close to 0 or changes slowly, the currentdepth pixel is determined to be located in a noise region if there is agreat difference between a depth value of the current depth pixel and adepth value of a neighboring depth pixel. Also, when a discontinuity ofthe gradient appears at a position of the current depth pixel, thecurrent depth pixel is determined to be located in an edge region of theobject.

The depth image refining apparatus adjusts the depth value of thecurrent depth pixel by applying a filter corresponding to the determinedtype of the region to the current depth pixel. The filter may be used toadjust the depth value of the current depth pixel based on the depthvalue of a neighboring depth pixel of the current depth pixel. A filtercharacteristic of the filter to be applied to the current depth pixelvaries depending on the type of region. Thus, the adjusting of the depthvalue of the current depth pixel based on the depth value of theneighboring depth pixel is referred to as spatial filtering. Relateddescriptions will be provided with reference to FIG. 3.

In another example, the depth image refining apparatus determineswhether to refine the depth image of a current time based on depthinformation of a depth image of a previous time. The depth imagerefining apparatus determines whether a change over time occurs in eachregion based on the change over time in the color image. For example,the depth image refining apparatus refines a depth image of the currenttime based on depth information of a depth image of the previous time inresponse to a difference between a color image corresponding to thedepth image of the current time and a color image corresponding to thedepth image of the previous time satisfying a preset condition. Therefining of the depth image of the current time based on the depthinformation of the depth image of the previous time is referred to astemporal filtering. The term “depth image of a current time” refers to adepth image that is currently being processed by the depth imagerefining apparatus to obtain a refined depth image. The term “depthimage of a previous time” refers to a depth image that was previouslyprocessed by the depth image refining apparatus to obtain a refineddepth image. In one example, the temporal filtering includes replacingdepth information of the depth image of the current time with the depthinformation of the depth image of the previous time. In another example,the temporal filtering includes replacing the depth information of thedepth image of the current time with a weighted sum of the depthinformation of the depth image of the current time and the depthinformation of the depth image of the previous time. However, these aremerely examples, and other types of temporal filtering may be used.

The temporal filtering may be performed for each region of the depthimage. In one example, a change over time in the color image does notoccur in a first region of the depth image, but a change over time inthe color image due to a movement of an object occurs in a second regionof the depth image. In this example, the temporal filtering is performedon a depth pixel included in the first region, but the temporalfiltering is not performed on a depth pixel included in the secondregion. In one example, the temporal filtering may be performed on adepth pixel included in a region of the depth image in response to adifference between a corresponding region of the color image of acurrent time and the corresponding region of the color image of aprevious time being less than a predetermined threshold. However, thisis just one example, and other conditions for determining when temporalfiltering is to be performed may be used instead of, or in addition to,this example.

In addition, both temporal filtering and spatial filtering may beapplied to the depth image. For example, the spatial filtering insteadof the temporal filtering may be applied to the second region, and thetemporal filtering as well as the spatial filtering may be applied tothe first region.

The depth information included in the depth image is refined to have amore accurate value based on the above-described refining process.

FIG. 3 is a flowchart illustrating an example of a process of refining adepth image based on surface normal information.

Referring to FIG. 3, in operation 310, the depth image refiningapparatus determines a type of a region to which a depth pixel includedin a depth image belongs based on surface normal information. The depthimage refining apparatus determines a region to which a current depthpixel belongs, for example, a noise region, a surface region of anobject, or an edge region of the object, based on a change in surfacenormal values corresponding to positions of the current depth pixel andneighboring depth pixels. As similarly described with reference to FIG.2, for example, when a surface normal value is constant or changesslowly, a current depth pixel is determined to be located in a surfaceregion (flat surface or curved surface) of an object. When the surfacenormal value is constant and then a discontinuity appears at a positionof the current depth pixel, the current depth pixel is determined to belocated in an edge region of the object. Based on the surface normalvalue, when a difference between a depth value of the current depthpixel and a depth value of a neighboring depth pixel is great eventhough the current depth pixel is not located in the edge region of theobject, the current depth pixel is determined to be located in a noiseregion.

In operation 320, the depth image refining apparatus adjusts a depthvalue of the depth pixel by applying a filter to the depth pixel. Thedepth image refining apparatus performs adaptive filtering using afilter corresponding to the type of the region to which the currentdepth pixel belongs. A filter characteristic of a filter, for example,either one or both of a filter size and a filter coefficient, to beapplied to the current depth pixel is determined based on a surfacenormal distribution for each region in the depth image.

In one example, a filter having a filter size that varies depending on adegree of change of the surface normal is applied to the current depthpixel. A filter having a relatively large filter size is applied to adepth pixel of a region in which a change in a surface normal is absentor small, and a filter having a relatively small filter size is appliedto a depth pixel of a region in which the change in a surface normal isgreat or a discontinuity appears. For example, the filter size to beapplied to a depth pixel of a noise region or a surface region of anobject is greater than a filter size to be applied to a depth pixel ofan edge region of the object.

In another example, a filter coefficient to be applied to a neighboringdepth pixel of the current depth pixel varies between a filter to beapplied to the current depth pixel of the noise region or the surfaceregion of the object and a filter to be applied to the current depthpixel of the edge region of the object. The filter coefficientcorresponds to a weight to be applied to a depth value of a neighboringdepth pixel in adjusting the depth value of the current depth pixel. Inone example, as the filter coefficient to be applied to the neighboringdepth pixel increases, an influence of the depth value of theneighboring depth pixel increases in determining the depth value of thecurrent depth pixel, and as the filter coefficient to be applied to theneighboring depth pixel decreases, an influence of the depth value ofthe neighboring depth pixel decreases in determining the depth value ofthe current depth pixel. For example, a filter coefficient to be appliedto a neighboring depth pixel in a filter to be applied to a currentdepth pixel of an edge region of an object is less than a filtercoefficient to be applied to a neighboring depth pixel in a filter to beapplied to a current depth pixel of a noise region or a surface regionof the object.

By performing the above-described adaptive filtering, a characteristicof the edge region of the object in the depth image may be maximallymaintained and a noise may be effectively removed based on the depthvalue of the neighboring depth pixel.

In operation 330, the depth image refining apparatus determines whetherthe current depth pixel is a last depth pixel of the depth image. Inresponse to the current depth pixel being the last depth pixel, theprocess of FIG. 3 terminates. In response to the current depth pixel notbeing the last depth pixel and other depth pixels to be processedremaining, operations 310 and 320 are performed on a next depth pixel.The depth image refining apparatus refines the depth image by performingoperations 310 and 320 on each depth pixel included in the depth image.

FIG. 4 illustrates an example of a process of refining a depth image.

Referring to FIG. 4, a depth image refining apparatus extracts shadinginformation from a color image 410 in operation 420, and estimatessurface normal information of an object surface from the shadinginformation. The depth image refining apparatus analyzes a surfacenormal distribution for each region in the color image 410 based on thesurface normal information in operation 430. The depth image refiningapparatus determines a way a surface normal changes for each region inthe color image 410. The depth image refining apparatus removes a noisefrom a depth image 415 corresponding to the color image 410 based on thesurface normal in operation 440. The depth image refining apparatusdetermines whether a depth pixel of the depth image 415 is located in anoise region, an edge region, or a surface region of the object based ona change in the surface normal, and reduces a noise component byapplying to the depth pixel a filter corresponding to a region in whichthe depth pixel of the depth image 415 is located. When theabove-described process is performed on each depth pixel of the depthimage 415, a refined depth image 450 is obtained.

In a case in which the depth value of the current depth pixel isreplaced with an average value of depth values of neighboring depthpixels, instead of performing adaptive filtering based on a region towhich the depth pixel belongs as disclosed in this application, both anoise component and an edge component of the object may be weakened. Inaddition, the depth value may be insufficient to distinguish between thenoise component and the edge component. However, the examples of thedepth image refining apparatus disclosed in this application distinguishbetween properties of regions to which depth pixels belong, and removethe noise component from the depth image 415 while maintaining the edgecomponent of the depth image 415 using a filter adapted for a propertyof a region.

FIG. 5 illustrates another example of a process of refining a depthimage.

Referring to FIG. 5, a depth image refining apparatus extracts an albedocomponent 520 and a shading component 530 from a color image 510. Thealbedo component 520 represents a unique color component of an object,and the shading component 530 represents a change of illumination valuescaused by a change in a surface normal of the object. The depth imagerefining apparatus estimates the change in the surface normal based onthe shading component 530. A change of shading values corresponds to achange of surface normal values. The depth image refining apparatusdetermines a type of a region to which each of depth pixels belongs in adepth image 540 based on the change of the surface normal values, andrefines the depth image 540 using a filter corresponding to thedetermined type of the region.

In the depth image 540, a region 544 is a region in which a change inthe shading component 530 is relatively small, and a type of the region544 is determined to be a surface region of the object. The depth imagerefining apparatus applies, to the depth pixels included in the region544, a filter for minimizing a noise other than a characteristic of adepth value distribution. For example, the depth image refiningapparatus applies, to the depth pixels included in the region 544, afilter having a relatively large filter size, or a filter for performingfiltering by greatly reflecting depth values of neighboring pixels. As afilter size increases, a number of neighboring depth pixels used toadjust the depth value of the current depth pixel increases. A region542 is a region in which the shading component 530 changes sharply, anda type of the region 542 is determined to be an edge region of theobject. The depth image refining apparatus applies, to depth pixels ofthe region 542, a filter having a relatively small filter size or afilter for performing filtering by less reflecting depth values ofneighboring depth pixels to maximally maintain a depth characteristic(edge characteristic) of the depth pixels included in the region 542.

In response to the above-described process of refining the depth pixelbeing performed on each depth pixel of the depth image 540, a refineddepth image 550 is obtained.

FIGS. 6A and 6B illustrate examples of determining a type of a region towhich a depth pixel belongs based on shading information.

Referring to FIG. 6A, a reference numeral 610 indicates depth values ofdepth pixels based on a position (x, y), and a reference numeral 630indicates a change in a shading value based on the position (x, y). Thechange in the shading value reflects a change in a surface normal of anobject. For example, in response to a shading value being constant in aregion, the region is estimated to be a surface of an object. Inresponse to the shading value being constant and then a discontinuityappearing in a region, a sharp change in a surface normal is located inthe region and the region is estimated to be an edge region of theobject. In response to the shading value changing (increasing ordecreasing) slowly in a region, the region is estimated to be a curvedsurface of the object.

In FIG. 6A, because a shading value of a region corresponding to a depthpixel 622 is constant based on the position (x, y), the depth pixel 622is estimated to be located in a surface region of an object. However,because a difference between a depth value of the depth pixel 622 anddepth values of neighboring depth pixels is relatively great, the depthvalue of the depth pixel 622 is estimated to be a noise component. Thedepth image refining apparatus performs filtering on the noise componentby applying a filter that sets weights of the neighboring depth pixelsto be great to the depth value of the depth pixel 622 estimated to bethe noise component. The depth image refining apparatus may apply afilter having a relatively large filter size to the depth value of thedepth pixel 622.

In a case of a depth pixel 624, a discontinuity of a shading valueappears at a position corresponding to the depth pixel 624, and thus thedepth pixel 624 is estimated to be located in an edge region of theobject. The depth image refining apparatus maintains an edge componentby applying a filter that sets weights of neighboring depth pixels to besmall to the depth value of the depth pixel 624. The depth imagerefining apparatus may apply a filter having a relatively small filtersize to the depth pixel 624.

Referring to FIG. 6B, a reference numeral 640 indicates depth values ofdepth pixels based on a position (x, y), and a reference numeral 660indicates a change in a shading value based on the position (x, y). Inone example, the depth image refining apparatus refines a depth imagebased on a window region 650. The window region 650 is a unit region forrefining the depth image. A size and a shape of the window region 650may vary. The depth image refining apparatus removes a noise from thedepth image by analyzing a correlation between surface normalinformation and depth information for each window region 650.

As indicated by the reference numeral 640, a noise component and an edgecomponent both may be present in the window region 650. In this case,the depth image refining apparatus effectively separates the noisecomponent and the edge component from depth information of depth pixelsincluded in the window region 650 by globally analyzing surface normalvalues in the window region 650. As indicated by the reference numeral660, because A and C have a same shading value and B and D have a sameshading value, A and C, and B and D may be estimated to have samesurface normal values. Thus, a distribution of depth values of depthpixels corresponding to A may be similar to a distribution of depthvalues of depth pixels corresponding to C, and a distribution of depthvalues of depth pixels corresponding to B may be similar to adistribution of depth values of depth pixels corresponding to D. Thus, adepth pixel corresponding to the noise component and a depth pixelcorresponding to the edge component may be easily separated from eachother in the window region 650.

FIGS. 7A and 7B illustrate examples of adjusting a depth value of adepth pixel using a filter.

Referring to FIG. 7A, a depth image 710 includes a plurality of depthpixels 720. A change in a surface normal in a color image is estimatedbased on shading information extracted from the color image, and a typeof a region to which a current depth pixel 722 belongs is determinedbased on the change in the surface normal. A depth image refiningapparatus applies to the current depth pixel 722, a filter having afilter size that varies depending on the type of region to which thecurrent depth pixel 722 belongs. In one example, the depth imagerefining apparatus applies a 3×3 (depth pixel unit) filter 732, a 5×5filter 734, or a 7×7 filter 736 to the current depth pixel 722 based onwhether the current depth pixel 722 belongs to an edge region of anobject, a surface region of the object, or a noise region, respectively.

In another example, a reduction rate of a filter coefficient isdetermined differently depending on the type of region to which thecurrent depth pixel 722 belongs. The filter coefficient decreases from acenter toward a periphery of the filter based on the reduction rate ofthe filter coefficient. For example, in response to the current depthpixel 722 belonging to an edge region, a reduction rate of a filtercoefficient to be applied to the current depth pixel 722 is greatcompared to reduction rates to be applied to other regions. Thus, arefined depth value of the current depth pixel 722 is less affected bydepth values of neighboring depth pixels. In response to the currentdepth pixel 722 belonging to a noise region, a reduction rate of afilter coefficient is small compared to reduction rates to be applied toother regions. Thus, the refined depth value of the current depth pixel722 is greatly affected by the depth values of the neighboring depthpixels. In response to the current depth pixel 722 belonging to asurface region, a reduction rate of a filter coefficient is intermediatebetween the reduction rate to be applied to the edge region and thereduction rate to be applied to the noise region. Thus, the refineddepth value of the current depth pixel 722 is moderately affected by thedepth values of the neighboring depth pixels. The sizes of a filterapplied to the noise region, a filter applied to the surface region, anda filter applied to an edge region may be the same, or may be adaptivelyvaried to be different based on a noise intensity.

FIG. 7B illustrates an example of adjusting a depth value of a depthpixel using a filter. The filter is used to adjust a depth value of acurrent depth pixel based on depth value of neighboring depth pixels. Areference numeral 740 indicates a filter to be applied to the currentdepth pixel, and a reference numeral 742 indicates depth information ofa region (region having a current depth pixel as a center) to which thefilter is to be applied. The filter is applied to a depth value e of acurrent depth pixel. A reference numeral 744 indicates a depth value e′of the current depth pixel adjusted by the filter.

For example, the depth value e′ of the current depth pixel adjusted bythe filter is determined using Equation 1 below.e′=(A×a)+(B×b)+(C×c)+(D×d)+(E×e)+(F×f)+(G×g)+(H×h)+(I×i)  (1)

In Equation 1, A, B, C, D, E, F, G, H, and I denote filter coefficients,and a, b, c, d, f, g, h, and i denote depth values of neighboring depthpixels of a current depth pixel e. Based on Equation 1, filtering isperformed based on depth information of a region to which a filter is tobe applied.

FIG. 8 is a flowchart illustrating an example of a process of extractingshading information using feedback.

Referring to FIG. 8, a depth image refining apparatus extracts shadinginformation from a color image based on the above-described process inoperation 810, and refines a depth image based on surface normalinformation of an object included in the extracted shading informationin operation 820. The depth image refining apparatus more accuratelyextracts the shading information from the color image based on depthinformation of the refined depth image. The depth image refiningapparatus estimates a region corresponding to a surface of the object ina window region based on a distribution of depth values of the refineddepth image. The depth image refining apparatus obtains more accurateshading information by increasing a weight of a shading component anddecreasing a weight of an albedo component in the region correspondingto the surface of the object so that, for example, the weight of theshading component is set to be greater than the weight of the albedocomponent. The refining process may be repeated until a desired degreeof accuracy is obtained.

Because a shading value does not greatly change in a surface region ofthe object, a sharp change in the shading value in the region estimatedto be the surface of the object is likely to be due to a change in thealbedo component. Based on the distribution of depth values of therefined depth image, the surface region of the object is identified, anda weight of the shading component is set to be greater than a weight ofthe albedo component for the surface region of the object in which thesharp change in the shading value occurs. The color of color pixels ofthe color image is determined by a product of the shading component andthe albedo component, and thus an influence of the albedo componentappearing in the surface region of the object on the shading informationis reduced by setting the weight of the shading component to be greaterthan the weight of the albedo component. Thus, more accurate shadinginformation is obtained.

FIG. 9 is a flowchart illustrating another example of an operation of amethod of refining a depth image.

Referring to FIG. 9, in operation 910, a depth image refining apparatusextracts shading information and albedo information from a color image.As described in operation 210 of FIG. 2, the depth image refiningapparatus extracts a shading component from the color image. The shadinginformation includes surface normal information of an object surface.After the shading component been extracted, the albedo component may beextracted by dividing a color of the color image by the shadingcomponent.

In operation 920, the depth image apparatus refines a depth image basedon a first weight based on the surface normal information (or shadinginformation), a second weight based on the albedo information, and athird weight based on a difference between a color image of a currenttime and a color image of a previous time. In one example, a localregression analysis is used to determine which one of the first weight,the second weight, and the third weight is an optimal weight for eachregion of the depth image. In one example, the optimal weight is alargest weight for each region among the first weight, the secondweight, and the third weight.

The depth image refining apparatus refines a depth value of a depthpixel based on a method that varies depending on the optimal weightdetermined for each region of the depth image. In one example, the firstweight is a largest weight among the first weight, the second weight,and the third weight and is determined to be the optimal weight in aregion in which a surface normal distribution is easily identifiedbecause the shading component is relatively accurately separated. Inthis example, the depth image refining apparatus determines a type of aregion, for example, an edge region of an object, a surface region ofthe object, or a noise region, to which a current depth pixel belongsbased on a change in a surface normal based on the above-describedmethod with respect to depth pixels included in the region, and appliesa filter having a filter characteristic that varies depending on thedetermined type of the region.

In another example, the second weight is a largest weight among thefirst weight, the second weight, and the third weight and is determinedto be the optimal weight in a region in which the albedo component isconstant even though the shading component is inaccurately separated. Inthis case, the depth image refining apparatus determines a type of aregion, for example, an edge region of an object, a surface region ofthe object, or a noise region, to which a current depth pixel belongsbased on the albedo information with respect to depth pixels included inthe region, and applies a filter having a filter characteristic thatvaries depending on the determined type of the region.

In another example, the third weight is a largest weight among the firstweight, the second weight, and the third weight and is determined to bethe optimal weight in a region in which a change or a difference betweena color image of a current time and a color image of a previous time isabsent or nearly absent. In this case, the depth image refiningapparatus adjusts the depth value based on depth information of a depthimage of the previous time with respect to the depth pixels included inthe region.

FIG. 10 illustrates an example of a configuration of a depth imagerefining apparatus.

Referring to FIG. 10, a depth image refining apparatus 1000 includes asensor 1010, a processor 1020, and a memory 1030. The sensor 1010, theprocessor 1020, and the memory 1030 communicate with each other via acommunication bus 1040. In one example, the sensor 1010 is providedoutside the depth image refining apparatus 1000, and the depth imagerefining apparatus 1000 receives image information from the sensor 1010.

The sensor 1010 includes an image sensor configured to obtain a colorimage and a depth sensor configured to obtain a depth image. In oneexample, the depth image refining apparatus 1000 includes both the imagesensor and the depth sensor, and in another example, the depth imagerefining apparatus 1000 includes only an image sensor of a stereo camerathat photographs a stereo image. In this last example, the color imageand the depth image are obtained from the stereo image photographed bythe stereo camera (through stereo matching). The sensor 1010 transfersthe obtained color image and depth image to either one or both of theprocessor 1020 and the memory 1030.

The processor 1020 performs either one or both of above-describedoperations of controlling the depth image refining apparatus 1000 andrefining the depth image. In one example, the processor 1020 extractsshading information of color pixels from the color image, and refinesthe depth image corresponding to the color image based on surface normalinformation of an object included in the shading information. Forexample, the processor 1020 adjusts a depth value of a current depthpixel by applying a filter to a current depth pixel included in thedepth image. A filter characteristic of the filter to be applied to thecurrent depth pixel is determined based on a surface normal distributionfor each region in the depth image. The processor 1020 determines a typeof a region to which the current depth pixel belongs based on thesurface normal information, and adjusts the depth value of the currentdepth pixel by applying the filter corresponding to the determined typeof the region to the current depth pixel.

In another example, the processor 1020 extracts albedo information andsurface normal information of the color pixels from the color image, andrefines the depth image based on a first weight based on the surfacenormal information, a second weight based on the albedo information, anda third weight based on a difference between a color image of a currenttime and a color image of a previous time. The processor 1020 determineswhich of the first weight, the second weight, and the third weight isgreatest for each region in the depth image, and refines a depth valueof a depth pixel based on a method that varies depending on a greatestweight. Related descriptions were provided above with reference to FIG.9.

In addition, the processor 1020 performs any one or any combination ofany two or more of operations described with reference to FIGS. 1through 9, and detailed descriptions of these operations are omitted.

The memory 1030 stores information used in the above-described processof refining the depth image and information of a result. Also, thememory 1030 stores instructions to be executed by the processor 1020. Inresponse to the instructions stored in the memory 1030 being executed bythe processor 1020, the processor 1020 performs any one or anycombination of any two or more of the above-described operations.

The depth image refining apparatus 1000 may receive a user input throughan input/output apparatus (not shown), and output data based on a resultof refining the depth image or a refined depth image. In addition, thedepth image refining apparatus 1000 may be connected to an externaldevice (not shown), for example, a personal computer or a network,through a communication device (not shown), thereby performing a dataexchange.

The image sensor 110, the depth sensor 120, and the depth image refiningapparatus 130 in FIG. 1 and the depth image refining apparatus 1000, thesensor 1010, the processor 1020, the memory 1030, and the communicationbus 1040 in FIG. 10 that perform the operations described in thisapplication are implemented by hardware components configured to performthe operations described in this application that are performed by thehardware components. Examples of hardware components that may be used toperform the operations described in this application where appropriateinclude controllers, sensors, generators, drivers, memories,comparators, arithmetic logic units, adders, subtractors, multipliers,dividers, integrators, and any other electronic components configured toperform the operations described in this application. In other examples,one or more of the hardware components that perform the operationsdescribed in this application are implemented by computing hardware, forexample, by one or more processors or computers. A processor or computermay be implemented by one or more processing elements, such as an arrayof logic gates, a controller and an arithmetic logic unit, a digitalsignal processor, a microcomputer, a programmable logic controller, afield-programmable gate array, a programmable logic array, amicroprocessor, or any other device or combination of devices that isconfigured to respond to and execute instructions in a defined manner toachieve a desired result. In one example, a processor or computerincludes, or is connected to, one or more memories storing instructionsor software that are executed by the processor or computer. Hardwarecomponents implemented by a processor or computer may executeinstructions or software, such as an operating system (OS) and one ormore software applications that run on the OS, to perform the operationsdescribed in this application. The hardware components may also access,manipulate, process, create, and store data in response to execution ofthe instructions or software. For simplicity, the singular term“processor” or “computer” may be used in the description of the examplesdescribed in this application, but in other examples multiple processorsor computers may be used, or a processor or computer may includemultiple processing elements, or multiple types of processing elements,or both. For example, a single hardware component or two or morehardware components may be implemented by a single processor, or two ormore processors, or a processor and a controller. One or more hardwarecomponents may be implemented by one or more processors, or a processorand a controller, and one or more other hardware components may beimplemented by one or more other processors, or another processor andanother controller. One or more processors, or a processor and acontroller, may implement a single hardware component, or two or morehardware components. A hardware component may have any one or more ofdifferent processing configurations, examples of which include a singleprocessor, independent processors, parallel processors,single-instruction single-data (SISD) multiprocessing,single-instruction multiple-data (SIMD) multiprocessing,multiple-instruction single-data (MISD) multiprocessing, andmultiple-instruction multiple-data (MIMD) multiprocessing.

The methods illustrated in FIGS. 2-4 and 9 that perform the operationsdescribed in this application are performed by computing hardware, forexample, by one or more processors or computers, implemented asdescribed above executing instructions or software to perform theoperations described in this application that are performed by themethods. For example, a single operation or two or more operations maybe performed by a single processor, or two or more processors, or aprocessor and a controller. One or more operations may be performed byone or more processors, or a processor and a controller, and one or moreother operations may be performed by one or more other processors, oranother processor and another controller. One or more processors, or aprocessor and a controller, may perform a single operation, or two ormore operations.

Instructions or software to control computing hardware, for example, oneor more processors or computers, to implement the hardware componentsand perform the methods as described above may be written as computerprograms, code segments, instructions or any combination thereof, forindividually or collectively instructing or configuring the one or moreprocessors or computers to operate as a machine or special-purposecomputer to perform the operations that are performed by the hardwarecomponents and the methods as described above. In one example, theinstructions or software include machine code that is directly executedby the one or more processors or computers, such as machine codeproduced by a compiler. In another example, the instructions or softwareincludes higher-level code that is executed by the one or moreprocessors or computer using an interpreter. The instructions orsoftware may be written using any programming language based on theblock diagrams and the flow charts illustrated in the drawings and thecorresponding descriptions in the specification, which disclosealgorithms for performing the operations that are performed by thehardware components and the methods as described above.

The instructions or software to control computing hardware, for example,one or more processors or computers, to implement the hardwarecomponents and perform the methods as described above, and anyassociated data, data files, and data structures, may be recorded,stored, or fixed in or on one or more non-transitory computer-readablestorage media. Examples of a non-transitory computer-readable storagemedium include read-only memory (ROM), random-access memory (RAM), flashmemory, CD-ROMs, CD-Rs, CD+Rs, CD-RWs, CD+RWs, DVD-ROMs, DVD-Rs, DVD+Rs,DVD-RWs, DVD+RWs, DVD-RAMs, BD-ROMs, BD-Rs, BD-R LTHs, BD-REs, magnetictapes, floppy disks, magneto-optical data storage devices, optical datastorage devices, hard disks, solid-state disks, and any other devicethat is configured to store the instructions or software and anyassociated data, data files, and data structures in a non-transitorymanner and provide the instructions or software and any associated data,data files, and data structures to one or more processors or computersso that the one or more processors or computers can execute theinstructions. In one example, the instructions or software and anyassociated data, data files, and data structures are distributed overnetwork-coupled computer systems so that the instructions and softwareand any associated data, data files, and data structures are stored,accessed, and executed in a distributed fashion by the one or moreprocessors or computers.

While this disclosure includes specific examples, it will be apparentafter an understanding of the disclosure of this application thatvarious changes in form and details may be made in these exampleswithout departing from the spirit and scope of the claims and theirequivalents. The examples described herein are to be considered in adescriptive sense only, and not for purposes of limitation. Descriptionsof features or aspects in each example are to be considered as beingapplicable to similar features or aspects in other examples. Suitableresults may be achieved if the described techniques are performed in adifferent order, and/or if components in a described system,architecture, device, or circuit are combined in a different manner,and/or replaced or supplemented by other components or theirequivalents. Therefore, the scope of the disclosure is defined not bythe detailed description, but by the claims and their equivalents, andall variations within the scope of the claims and their equivalents areto be construed as being included in the disclosure.

What is claimed is:
 1. A method of refining a depth image, the methodcomprising: extracting shading information of color pixels from a colorimage; and refining a depth image corresponding to the color image basedon surface normal information of an object included in the shadinginformation, wherein the refining of the depth image includesdistinguishing between a noise region and an edge region in the depthimage based on a characteristic of a surface normal distributionrepresented in the color image and applying filters comprising differentfilter characteristics to the noise region and the edge regionrespectively to reduce a noise component while maintaining an edgecharacteristic substantially unchanged.
 2. The method of claim 1,further comprising extracting shading information of the color pixelsbased on depth information of the refined depth image.
 3. The method ofclaim 1, wherein the shading information corresponds to a vector dotproduct between the direction of the light source and the surface normalof the object surface.
 4. A non-transitory computer-readable mediumstoring instructions that, when executed by a processor, cause theprocessor to perform the method of claim
 1. 5. The method of claim 1,wherein the refining of the depth image comprises: determining thefilter characteristic of the filter to be applied to a current depthpixel, of the depth pixels, included in the depth image based on asurface normal distribution of the surface normal information for eachof regions in the depth image; and adjusting a depth value of thecurrent depth pixel by applying the filter having the determined filtercharacteristic to the current depth pixel.
 6. The method of claim 5,wherein the determining of the filter characteristic comprisesdetermining either one or both of a filter coefficient of the filter tobe applied to the current depth pixel and a filter size of the filter tobe applied to the current depth pixel based on the surface normalinformation.
 7. The method of claim 1, further comprising determiningwhether to refine a depth image of a current time based on depthinformation of a depth image of a previous time.
 8. The method of claim7, further comprising refining the depth image of the current time basedon the depth information of the depth image of the previous time inresponse to a difference between a color image corresponding to thedepth image of the current time and a color image corresponding to thedepth image of the previous time satisfying a preset condition.
 9. Amethod of refining a depth image, the method comprising: extractingshading information of color pixels from a color image; and refining adepth image corresponding to the color image based on surface normalinformation of an object included in the shading information, whereinthe refining of the depth image comprises: determining a type of aregion to which a current depth pixel included in the depth imagebelongs based on a surface normal distribution of the surface normalinformation in the region; and adjusting a depth value of the currentdepth pixel by applying a filter corresponding to the determined type ofthe region to the current depth pixel.
 10. The method of claim 9,wherein the filter is configured to adjust the depth value of thecurrent depth pixel based on a depth value of a neighboring depth pixelof the current depth pixel.
 11. The method of claim 9, wherein thedetermining of the type of the region comprises determining a region towhich the current depth pixel belongs based on a change of surfacenormal values of neighboring pixels of the current depth pixel.
 12. Themethod of claim 9, wherein the determining of the type of the regioncomprises determining whether the current depth pixel belongs to a noiseregion, a surface region of the object, or an edge region of the object.13. The method of claim 12, wherein a filter size of a filtercorresponding to the noise region or the surface region of the object isgreater than a filter size of a filter corresponding to the edge regionof the object.
 14. The method of claim 12, wherein a filter coefficientto be applied to a neighboring depth pixel of the current depth pixelvaries depending on whether the filter to be applied to the currentdepth pixel is a filter corresponding to the noise region, a filtercorresponding to the surface region of the object, or a filtercorresponding to the edge region of the object.
 15. A method of refininga depth image, the method comprising: extracting shading information ofcolor pixels from a color image; and refining a depth imagecorresponding to the color image based on surface normal information ofan object included in the shading information, further comprisingextracting albedo information of the color pixels from the color image;wherein the refining of the depth image comprises refining the depthimage based on a first weight based on the surface normal information, asecond weight based on the albedo information, and a third weight basedon a difference between a color image of a current time and a colorimage of a previous time.
 16. A depth image refining apparatuscomprising: a processor configured to: extract shading information ofcolor pixels from a color image, and refine a depth image correspondingto the color image based on surface normal information of an objectincluded in the shading information, wherein the refining of the depthimage includes distinguishing between a noise region and an edge regionin the depth image based on a characteristic of a surface normaldistribution represented in the color image and applying filterscomprising different filter characteristics to the noise region and theedge region respectively to reduce a noise component while maintainingan edge characteristic substantially unchanged.
 17. The depth imagerefining apparatus of claim 16, wherein the processor is furtherconfigured to: determine a filter characteristic of the filter to beapplied to a current depth pixel, of the depth pixels, included in thedepth image based on a surface normal distribution of the surface normalinformation for each of regions in the depth image, and adjust a depthvalue of the current depth pixel by applying the filter having thedetermined filter characteristic to the current depth pixel.
 18. Thedepth image refining apparatus of claim 16, wherein the processor isfurther configured to: determine a type of a region to which a currentdepth pixel, of the depth pixels, included in the depth image belongsbased on a surface normal distribution of the surface normal informationof the region, and adjust a depth value of the current depth pixel byapplying a filter corresponding to the determined type of the region tothe current depth pixel.
 19. The depth image refining apparatus of claim16, wherein the processor is further configured to: extract albedoinformation of the color pixels from the color image, and refine thedepth image based on a first weight based on the surface normalinformation, a second weight based on the albedo information, and athird weight based on a difference between a color image of a currenttime and a color image of a previous time.
 20. A method of refining adepth image, the method comprising: determining a noise reducing methodto be applied to a depth image based on surface normal information of anobject in a color image corresponding to the depth image; and refiningthe depth image by applying the determined noise reducing method to thedepth image, the determined noise reducing method includingdistinguishing between a noise region and an edge region in the depthimage based on a characteristic of a surface normal distributionrepresented in the color image and applying filters comprising differentfilter characteristics to the noise region and the edge regionrespectively to reduce a noise component while maintaining an edgecharacteristic substantially unchanged.
 21. The method of claim 20,wherein the color image is a color image of a current time; and themethod further comprises applying a temporal filter, as the filter, tothe current depth pixel in response to a difference between a region ofthe color image of the current time corresponding to a region of thedepth image in which the current depth pixel is located and acorresponding region of a color image of a previous time being less thana predetermined threshold.
 22. The method of claim 20, wherein thedetermining of the noise reducing method comprises determining a filtercharacteristic of the filter based on the surface normal information;and the refining of the depth image comprises applying the filter to acurrent depth pixel of the depth image.
 23. The method of claim 22,wherein the filter is a spatial filter comprising a filter coefficientto be applied to a depth value of the current depth pixel and filtercoefficients to be applied to depth values of neighboring depth pixelsof the current depth pixel to obtain a refined depth value of thecurrent depth pixel; and the determining of the filter characteristiccomprises determining, based on the surface normal information, eitherone or both of a filter size of the spatial filter and a reduction rateat which the filter coefficients decrease from a center of the spatialfilter to a periphery of the spatial filter.
 24. The method of claim 23,wherein the determining of the filter characteristic further comprises:determining whether a region in which the current depth pixel is locatedis a noise region, a surface region of the object, or an edge region ofthe object based on a surface normal distribution of the surface normalinformation; and determining either one or both of the filter size andthe reduction rate based on the determined type of the region.
 25. Themethod of claim 24, wherein the determining of the filter sizecomprises: determining the filter size to be a first filter size inresponse to the determined type of the region being the noise region;determining the filter size to be a second filter size smaller than thefirst filter size in response to the determined type of the region beingthe surface region of the object, and determining the filter size to bea third filter size smaller than the second filter size in response tothe determined type of the region being the edge region of the object;and the determining of the reduction rate comprises: determining thereduction rate to be a first reduction rate in response to thedetermined type of the region being the noise region; determining thereduction rate to be a second reduction rate greater than the firstreduction rate in response to the determined type of the region beingthe surface region of the object, and determining the reduction rate tobe a third reduction rate greater than the second reduction rate inresponse to the determined type of the region being the edge region ofthe object.